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Background / Context: 

Description of prior research and its intellectual context. 

The goal of many preschool interventions is to better prepare children for the academic 
demands of school. Curriculum-based interventions endeavor to improve the ways in which 
teachers deliver instruction and developers of curriculum interventions often base curriculum 
design on assumptions about teaching and learning, build these into the content and structure of 
the activities, and provide teachers guidance that is designed to help foster learning in ways 
consistent with the curriculum. Program directors and practitioners often select curricular 
packages based on researchers’ claims that they are effective in increasing children’s academic 
development. Yet, in order for a curriculum intervention to claim success, researchers and 
developers of the curriculum must be able to identify the essential instructional and structural 
components and recognize the specific goals of the curriculum. They must also ensure adequate 
levels of delivery and provide evidence that the treatment condition differs significantly from the 
control condition through the use of well-developed measures of implementation fidelity. 

Researchers often grapple with the multidimensional nature of teaching and learning and 
the inherent difficulties in implementing effective educational interventions. Models of 
interventions, as well as the fidelity measures developed to ensure teachers have adequately 
implemented the causal components, can be equally complex. Despite efforts to create effective 
measures of fidelity, the instruments produced by many curriculum developers and educational 
researchers and the methods in which they have been used have frequently fallen short. This 
paper focuses on the development and application of fidelity measures that represent the critical 
components of a preschool curriculum intervention. Data collected via these measures are 
presented through three approaches and provide evidence that fidelity of implementation is a 
complex and multi-dimensional concept. 

A clear, accurate conceptual model acts as a guide to researchers and developers when 
creating fidelity measures that assess teachers on the degree to which they implement the causal 
components of the intervention. Use of the measures to assess teachers on the degree to which 
and quality with which they deliver the curriculum enables researchers to interpret the 
effectiveness of the intervention. Results from analysis of fidelity of implementation also reveal 
the ways in which specific elements of the intervention were and were not delivered. This 
information allows researchers to modify professional development and other in-service training 
of teachers. In addition, a better understanding of the difficulties teachers face when 
implementing the curriculum drives researchers and developers to rethink the feasibility of the 
intervention. 

Purpose / Objective / Research Question / Focus of Study: 

Description of the focus of the research. 

This paper presents the development of reliable measures of fidelity that represent critical 
elements of the curriculum and its use in assessing teachers’ delivering of the treatment 
curriculum and identifying the existence of program differentiation between the treatment and 
control conditions. In order to address several issues that arise when assessing levels of 
implementation while confirming differentiation between treatment and control conditions 
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through the use of well-developed measures, fidelity data eollected from a presehool eurrieulum 
intervention have been analyzed to demonstrate an example of: 

• Creating psyehometrieally strong, eomprehensive fidelity measures that represent the 
eritical components of an intervention 

• Using those measures to view fidelity as something beyond a unitary construct by 
differentiating implementation of instructional constructs and activities across conditions 

Setting: 

Description of the research location. 

Data in this study were collected as part of Teacher Enhanced Language and Literacy 
(TELL)^ that involved a large Head Start program in a medium sized southern city. In this four- 
year intervention, teachers implemented the Opening the World of Learning (OWL) curriculum 
(Schickedanz & Dickinson, 2005) in conjunction with The Creative Curriculum (Dodge, Colker, 
& Heroman, 2002) in 36 treatment classrooms. Teachers in sixteen additional classrooms used 
The Creative Curriculum exclusively as the primary curriculum (i.e. business as usual). 

Intervention / Program / Practice: 

Description of the intervention, program or practice, including details of administration and duration. 

OWL is a comprehensive curriculum with emphasis on developing language and literacy 
skills in preschool children. The curriculum spans a full year and is separated into six units 
focused on themes ranging from Family to Things that Grow. The preschool day is organized 
into six activities: Morning Meeting, Centers, Small Group (SG), Book Reading (READ), Group 
Literacy Instruction (GLI), and Let’s Lind Out About It/Lef s Talk About It. 

Of all the activities included in the OWL curriculum and implemented by teachers 
assigned to the treatment condition, this study includes analysis of data obtained from (a) Small 
Group, (b) Book Reading, and (c) Group Literacy Instruction. These three activities in particular 
are designed to increase children’s exposure to concepts of language and literacy considered to 
be integral to their development. Because of this, it is important to understand how well and to 
what extent teachers are implementing lessons within these three activities. 

Research Design: 

Description of research design (e.g, qualitative case study, quasi-experimental design, secondary analysis, analytic 
essay, randomized field trial). 

Analyses reported in this presentation come from a larger lES-funded Randomized 
Controlled Trial (RCT) in which clusters of Head Start programs were randomly assigned to 
treatment or control conditions. 

Data Collection and Analysis: 

Description of the methods for collecting and analyzing data. 



^ The research reported here was supported by the Institute of Education Sciences, U.S. 
Department of Education, through Grant R324E060088A to Vanderbilt University. 
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Three aetivities were videotaped at two time points during the sehool year for eaeh of the 
52 elassrooms. Observers eoded two sessions of eaeh videotaped aetivities for teaehers’ fidelity 
of implementation. Seores from these two sessions were then averaged to ereate a elassroom 
seore. 

Instruments were created to assess teachers on the frequency with which they 
implemented essential elements of the curriculum across the day. A series of checklists was 
developed to represent three daily activities: Book Reading, Small Groups, and GLI. With the 
guidance of the co-author of OWL, instructional items identified as uniquely related to the 
objectives of OWL were identified. The number of OITL-specific items varied across activities 
with Book Reading, Small Group, and GLI checklists containing 9, 8, and 7 items respectively. 

Findings / Results: 

Description of the main findings with specific details. 

Creating a Psychometrically Strong, Comprehensive Fidelity Measure 

This fidelity measure (represented in Figure 1) accounts for three levels of the 
intervention: (1) curriculum-level, (2) activity-level, and (3) instructional- and structural- 
subconstruct level. The curriculum-level of the model includes the three primary daily activities: 
Small Group, GLI, and Book Reading. The activity-level of the model includes measures of 
process- and compliance directly related to each activity. Lastly, the instructional- and 
structural-subconstruct level contains groups of fidelity indicators contained in the observational 
checklist for that activity (e.g., SI refers to the first item in Small Group Checklist). There are 
some commonalities in the subconstructs within each of these activities, yet the delivery of 
specific curricular elements is also unique to the different activities and settings. 

Internal consistency reliability was assessed by calculating a Cronbach’s coefficient alpha 
for all items within each activity and for all items grouped by instructional and structural 
subconstruct within each activity. Figure 1 shows that the overall scales for Small Group, GLI, 
and Book Reading activities were 0.77, 0.69, and 0.57 respectively. The calculated Cronbach’s 
alphas for instructional constructs within activities ranged from 0.19 to 0.78. Any construct with 
an internal consistency less than 0.40 was dropped from the final analysis. The results of internal 
consistency tests confirm that the majority of fidelity indicators reliably represent the critical 
elements contained within the conceptual model at both the activity- and subconstruct-level. 
Therefore, the fidelity measures used in this study provide a valid picture of implementation 
levels demonstrated by teachers. 

Using Measures to View Fidelity as Something beyond a Unitary Construct 

The comprehensive set of fidelity measures used in this study provided a nuanced 
understanding of implementation, revealing differential levels of implementation between 
teachers assigned to the treatment and control groups. Fidelity measures were used to evaluate 
teachers’ implementation through three different perspectives: overall fidelity across activities, 
fidelity by activity, and fidelity by structural and instructional subconstructs within activities. 

A curriculum-level view: overall curriculum implementation by condition. As 
expected, teachers assigned to implement OWL had significantly higher scores than control 
teachers in overall fidelity to OWL across curricular activities. Treatment teachers implemented 
50.2% (SD = 10.6) of the critical elements of OWL, averaged across two times and across three 
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curricular activities. This level of implementation was less than ideal, yet it was signifieantly 
higher than that of the eontrol teaehers (M = 27.5%; SD = 8.1). 

Zooming in: curriculum implementation at the activity-level. Unique patterns of 
implementation emerged, however, when examining levels of fidelity to OWL within speeifie 
eurrieular aetivities. Treatment teaehers implemented OlTL-speeifie elements of Small Group 
(M = 62.67) at higher rates than in any other aetivity. However, eontrol teaehers were also 
sueeessful at implementing elements of Small Group instruetion that were endorsed by OWL (M 
= 28.91) (see Table 1 for eomplete data). Despite delivering more than half of the eurrieulum- 
speeifie items in Small Group, treatment teaehers did not differ mueh from the teaehers assigned 
to the eontrol eondition, in whieh no OlTL-speeifie instruetion was expeeted. Thus, the 
apparently stronger levels of fidelity in the OWL elassrooms were negated by the faet that the 
instruetion delivered by treatment and eontrol teaehers in Small Groups was not as different as 
researehers predieted at the start of the study. 

Overall levels of fidelity by treatment teaehers during GLI were similar to Small Group. 
On average, treatment teaehers implemented roughly 53% of OlTL-speeifie items, yet eontrol 
teaehers delivered 21.4% of the OWL eurrieulum. Although this differenee between treatment 
and eontrol is statistieally signifieant {t = -5.90, /> = .000, DF = 50), these levels of 
implementation do not differ to the extent that researehers expeeted. Data also suggest that 
treatment teaehers had the most diffieulty in implementing OWL during Book Reading (M = 
32.1). Differenees between the ORX-speeifie instruetion delivered in the treatment elassrooms 
as eompared to the eontrol elassrooms were also at their lowest during Book Reading. 

The analysis of fidelity at the aetivity-level provides a more speeifie pieture of 
differentiation between treatment and eontrol elassrooms in how teaehers delivered instruetion 
within eaeh of the three eurrieular aetivities. Although rates of OWL implementation were 
signifieantly higher for treatment teaehers in all three aetivities, data revealed more similarities 
between eonditions than did the eurrieulum-level view of fidelity diseussed earlier. 

A nuanced view: analysis of structural and instructional subconstructs. A third and 
final analysis shows that patterns of implementation also varied with regard to speeifie 
instruetional and struetural elements highlighted in OWL. Beeause the same set of validated tools 
was used to observe and assess both treatment and eontrol teaehers on the degree to whieh they 
implemented partieular instruetional methods and struetural eomponents, it was possible to foeus 
on nuaneed differenees between the eonditions. Treatment teaehers showed higher rates of 
fidelity in most of the instruetional subeonstruets related to Small Group and GLI aetivities. In 
partieular, they were more supportive of ehildren’s language development and analytieal 
thinking in Small Group aetivities. In addition, treatment teaehers delivered eode-foeused 
instruetion during Group Literaey Instruetion with higher fidelity than did eontrol teaehers. 

The set of fidelity measures produeed unexpeeted results when data eolleeted from the 
Book Reading aetivity were analyzed. There were no statistieal differenees in the ways that 
treatment and eontrol teaehers supported ehildren’s language development and analytieal 
thinking while reading storybooks. 

Data eolleeted on teaeher fidelity also represented struetural elements of implementation, 
referred to as eomplianee in this study. In a similar way to teaehers’ implementation patterns of 
instruetional elements, teaehers’ levels of eomplianee to the treatment eurrieulum followed a 
mixed pattern. Treatment teaehers demonstrated greater eomplianee than eontrol teaehers during 
GLI aetivities in that they presented aetivities that were speeified by lesson plans for the 
reeommended length of time. On the other hand, treatment teaehers were statistieally equivalent 
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to control teachers when adhering to struetural elements of Small Group aetivities. Treatment 
teaehers did not eonsistently seleet the most ehallenging lessons seheduled for that day and 
engage ehildren for the reeommended length of time for Small Group aetivities. Lower rates of 
implementation may point to a laek of ehange in teaeher behavior, but it may also indieate 
weakness in the measure itself. 

Conclusions: 

Description of conclusions, recommendations, and limitations based on findings. 

The analyses presented in this paper provide evidenee that psyehometrieally valid set of 
measures may indeed be used to aeeount for the multi-dimensional nature of implementation 
fidelity. Given the results of the internal eonsisteney reliability testing, it is possible to eonelude 
that the ereation of aeeurate and valid models and assoeiated fidelity indieators is eomplex and 
ehallenging. Although this model does represent key eomponents of the eurrieulum, it has some 
limitations whieh will be diseussed during the eonferenee presentation. 

The eomprehensive and multidimensional set of fidelity measures used in this study 
provided researehers the opportunity to examine implementation through a variety of 
perspeetives. When defining fidelity as a unitary eonstruet by assigning eaeh teaeher a single 
value of fidelity aeross aetivities (i.e., at the eurrieulum-level), results suggest that treatment 
teaehers implemented the intervention eurrieulum at signifieantly higher rates than did eontrol 
teaehers. When examining levels of fidelity within partieular eurrieular aetivities, data revealed 
that the treatment teaehers were the better implementers in all three of the activities but levels of 
differentiation between eonditions varied by aetivity. Furthermore, this perspeetive provided 
evidenee that treatment teaehers had more diffieulty implementing Book Reading than any other 
aetivity. The final and most nuaneed analysis whieh reported fidelity at the instruetional- and 
struetural-subeonstruet level revealed a different pieture of fidelity by exposing instanees in 
whieh treatment teaehers did not implement instruetional and struetural elements unique to the 
intervention eurrieulum at higher rates than the eontrol teaehers. 

Views of implementation fidelity as a unitary eonstruet may obseure details about the 
speeifie ways in whieh treatment and eontrol teaehers vary and may limit how researehers 
understand eurrieulum interventions. The addition of fine-grained analyses made possible by 
multi-dimensional fidelity measures that examine the degree to whieh teaehers from different 
experimental eonditions deliver speeifie struetural and instruetional elements unique to the 
treatment eondition is neeessary to provide a robust and aeeurate representation of 
implementation. This analytieal approaeh has implieations for the ways in whieh edueational 
researehers pereeive the eomplexity of implementation fidelity, interpret intervention 
effeetiveness, and provide valuable professional development. 
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Small Groip Cronbach's Alpha = 0.82 



Groip Literacy Iistni^tion Cronbach's Alpha — 0.74 



Book Reaclliig Cronbachls Alpha = 0.51 



Figure 1. Change model for OWL eurrieulum intervention. 



Table 1 

Percentage Scores for Fidelity for each Curricular Activity by Condition 



Small Group GLI Book Reading 

Treatment Control Treatment Control Treatment Control 







Mean (SD) 




Mean (SD) 




Mean (SD) 


OWL 


62.7 


28.9 


52.6 


21.4 


32.1 


18.5 


FOI 


(20.6) 


(14.2) 


(18.5) 


(15.2) 


(11.4) 


(9.0) 


CC 


52.8 


54.5 


94.8 


80.5 


42.3 


51.3 


FOI 


(15.5) 


(20.5) 


(9.6) 


(21.4) 


(8.8) 


(13.6) 
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